Data Dropout in Arbitrary Basis for Deep Network Regularization

An important problem in training deep networks with high capacity is to ensure that the trained network works well with new inputs. Dropout is an effective regularization technique to boost the network’s generalization in which a random subset of the elements of given data and the extracted features are set to zero during the training process. In this paper, a new randomized regularization technique is proposed in which we withhold a random part of data without necessarily turning off the neurons/data-elements. In the proposed method, of which the conventional dropout is shown to be a special case, random data dropout is performed in an arbitrary basis. In addition, we present a framework to efficiently apply the proposed technique to the convolutional neural networks. The presented numerical experiments show that the proposed technique yields notable performance gain. The proposed approach, dubbed Generalized Dropout, provides a deep insight into the idea of dropout, shows that we can achieve different performance gains using different basis matrices, and opens up a new research question as of how to choose optimal basis matrices that achieve maximal performance gain.